The present invention relates to an image sensing apparatus and, more particularly, to an image sensing apparatus, using a non-interlace scanning type image sensing device, capable of encoding image data obtained by the image sensing device in accordance with movement of an object sensed, and outputting a smooth moving image when reproducing image data recorded on a recording medium.
Recently, a non-interlace scanning type image sensing device capable of sequentially reading signals of all the pixels has been developed with the progress of semiconductor manufacturing technique.
The non-interlace scanning type image sensing device has an advantage in that a higher resolution image can be obtained with less blurring than an image sensed by using a conventional interlace scanning type image sensing device even when sensing a moving object.
In the interlace scanning type image sensing device, a frame image is composed of two field images which are sensed at different times, usually at a field period interval. Accordingly, there is a problem in which, when sensing a fast moving object, there are notches on edges of the object and perhaps of the background in a frame image because of the time gap between the two field images composing a frame image.
If a frame image is made of image data of a single field image to overcome the aforesaid problem, there would not be notches on edges, however, since the amount of image information in the vertical direction is halved compared to a frame image composing of two field images, the vertical resolution of the obtained frame image is also halved.
In contrast, with a non-interlace scanning type image sensing device, it is possible to sense a frame image in the same time period as that for sensing a field image by an interlace scanning type image sensing device, thus, the above problem does not arise. By taking this advantage of the non-interlace scanning type image sensing device, it is applied to a still image camera and an input device for use with a computer, for example.
Further, in a still image output device, such as a video printer, which has rapidly spread in the market in these days, a user can arbitrary pick up a desired scene out of images which are sensed as a moving image. Accordingly, there is a demand to use the non-interlace scanning type image sensing device as an image sensing unit of a video camera capable of sensing both a moving image and a still image.
When a non-interlace scanning type image sensing device is used as an image sensing unit of a video camera capable of sensing both a moving image and a still image, as described above, a couple of methods for generating moving image signals can be considered.
A case where a non-interlace scanning type image sensing device is used in a digital video camera of NTSC standard, as shown in FIG. 7, will be explained as an example. A non-interlace scanning type image sensing device 1 has a structure to output signals by two channels, and each channel always outputs either image signals of even lines or image signals of odd lines of the non-interlace type image sensing device 1.
Further, in FIG. 7, image signals of the even lines and odd lines are alternatively outputted from each channel of the non-interlace scanning type image sensing device 1 in each field period in accordance with timing signals generated by a timing signal generator (TG) 15. For example, referring to one of the two output channels, when image signals of even lines are outputted from one of the channel in a given field period, image signals of odd lines are outputted in the next field period, then image signals of even lines are outputted in the following field period. Image signals read out from the non-interlace scanning type image sensing device 1 are respectively inputted to correlated double sampling (CDS) circuits 201 and 202. The signals outputted from the CDS circuits 201 and 202 are inputted to automatic gain controllers (AGCs) 301 and 302, thereafter enter analog-digital (A/D) converters 401 and 402, respectively.
Then, after the analog signals are converted into digital signals by the A/D converters 401 and 402, enter a camera signal processing circuit 5. The camera signal processing circuit 5 performs signal processes, such as color separation, edge enhancement, and color correction, after the image data of the even lines and odd lines are applied with dot sequential processing.
After the aforesaid processes are completed, the camera signal processing circuit 5 divides a frame image, and image data of one field (e.g., image data of even lines) is outputted from the first channel ch1, and image data of the other field (e.g., image data of odd lines) is outputted from the second channel ch2. Similarly, for the next frame image, image data of alternate fields are outputted from the first and second channels ch1 and ch2. For example, image data of odd lines is outputted from the first channel ch1, and image data of even lines is outputted from the second channel ch2.
The non-interlace scanning type image sensing device 1 can generate a frame image in one field period, however, a recording device (e.g., a digital VTR) can record only a field image in one field period.
Accordingly, as shown in FIG. 8A, by using either the image signals outputted from the first channel ch1 or the image signals outputted from the second channel ch2, an image of a single field is outputted in each field period (a mode for performing the aforesaid operation is called “field image sensing mode”, hereinafter).
Referring to FIG. 8A, the camera signal processing circuit 5 sequentially writes field image data of a first frame image #1 and of a second frame image #2 outputted from the first channel ch1 to a first frame memory 601 in the first and second field periods in accordance with a control signal C1.
Meanwhile, if image data is written in a second frame memory 602, the image data of one previous frame period is sent to an encoding processing circuit 7 in accordance with a control signal C2. In the encoding processing circuit 7, the image data is applied with processes, such as discrete cosine transform (DCT) and shuffling, thereafter, stored in a magnetic tape 9 by a recording head 8 as digital image signals in a recording method complying with a format.
After image data of one frame is written in the first frame memory 601, the camera signal processing circuit 5 sequentially writes field image data of the third frame image #3 and of the fourth frame image #4 outputted from the first channel ch1 to the second frame memory 602 in the third and fourth field periods in accordance with the control signal C2.
Meanwhile, the image data of one previous frame period is sent to the encoding processing circuit 7 in accordance with a control signal C2. In the encoding processing circuit 7, the image data is applied with the same processes as described above, thereafter, stored in the magnetic tape 9 by the recording head 8 as digital image signals in the recording method complying with the format.
The image signals which are recorded as above is processed as shown in FIG. 8B when they are reproduced. First, the image signals read from the magnetic tape 9 by the read head 10 are sent to a decoding processing circuit 11 in the first and second field periods, and applied with signal processes, such as inverse discrete cosine transform (I-DCT) and de-shuffling. Thereafter, the image signals are written to a third frame memory 121 in accordance with a control signal C3.
Meanwhile, if image data is written in a fourth frame memory 122, the decoding processing circuit 11 reads image data from the fourth memory 122 in accordance with a control signal C4, and reproduces an image by an even line field and an odd line field separately.
In the third and fourth field periods, the image signals read from the magnetic tape 9 by the read head 10 are transmitted to the decoding processing circuit 11 where the image signals are applied with the I-DCT, deshuffling, and so on, then written to the fourth frame memory 122.
Meanwhile, the decoding processing circuit 11 reads image data from the third memory 121 in accordance with the control signal C3, and reproduces an image of the even line field and the odd line field separately.
In the decoding processing circuit 11 used in the aforesaid conventional example, image signals outputted from the first channel ch1 and the second channel ch2 are of the even line field and the odd line field of the non-interlace scanning type image sensing device 1, shown in FIG. 10. Therefore, the field image sensing mode can be used for moving image sensing operation.
However, according to the aforesaid example, the vertical resolution is about the same as that of a conventional image sensing device which outputs image data after adding charges stored in two adjacent pixels in the vertical direction. Accordingly, when outputting an image of a moving object obtained by using the non-interlace scanning type image sensing device 1 in the field image sensing mode as a still image from a video printer, or the like, only a poor still image can be obtained because of blurring.
Thus, as a method of effectively using an advantage of the non-interlace scanning type image sensing device, i.e., to output a frame image in a field period, the one shown in FIG. 9A has been suggested.
In this case, the camera signal processing circuit 5 writes field image data of the even line field of the frame image #1 outputted from the first channel ch1 and field image data of the odd line field of the frame image #1 outputted from the second channel ch2 to the first frame memory 601 in the first and second field periods, respectively, in accordance with the control signal C1.
Then a frame image #2 sensed in the second field period is not stored, and the first frame image sensed in the first field period is used as a moving image of a frame period (this image sensing operation is called “frame image sensing mode”, hereinafter).
Meanwhile, if image data has been written in the second frame memory 602, the image data of one previous frame is sent to the encoding processing circuit 7 in accordance with the control signal C2 and processed with DCT, shuffling, and so on. Thereafter, the processed image data is recorded as digital image signals on the magnetic tape 9 by the recording head 8 in a recording method complying with a format.
When image data of a single frame is written in the first frame memory 601, the camera signal processing circuit sequentially writes even line field image data of the third frame image #3 outputted from the first channel ch1 and odd line field image data of the third frame image #3 outputted from the second channel ch2 to the second frame memory 602 in the third and fourth field periods in accordance with the control signal C2.
Meanwhile, the image data of one previous frame written in the first frame memory 601 is send to the encoding processing circuit 7 where it is applied with the same processes as described above, then recorded on the magnetic tape 9 by the recording head 8 in a recording method complying with a format as digital image signals.
The image signals recorded as described above are applied with processes as shown in FIG. 9B when they are reproduced. First, the image signals read from the magnetic tape 9 by the read head 10 are transmitted to the decoding processing circuit 11 where they are processed with I-DCT, deshuffling, and so on. Then, the image signals are written to the third frame memory 121 in accordance with the control signal C3 in the first and second field periods.
Meanwhile, if image data is written in the fourth frame memory 122, the decoding processing circuit 11 reads image data from the fourth memory 122 in accordance with the control signal C4, and reproduces an image by an even line field and an odd line field separately.
In the third and fourth field periods, the image signals read from the magnetic tape 9 by the read head 10 are transmitted to the decoding processing circuit 11 where they are applied with the I-DCT, deshuffling, and so on. Thereafter, the images are written to the fourth frame memory 122 in accordance with the control signal C4.
Meanwhile, the decoding processing circuit 11 reads image data from the third memory 121 in accordance with the control signal C3, and reproduces an image using the even line field and the odd line field separately.
In this method, it is possible to store an image of high resolution without blurring when sensing a moving object. Therefore, the method can be used when sensing a still image. However, when the stored image data is reproduced as a moving image, image data of fields as shown in FIG. 9B is outputted, thus the displayed image is of frame images sensed in frame period. In this method, therefore, it is possible to obtain an image of high resolution, however, when the image data of a faster moving object is reproduced as a moving image, for example, the displayed image has gaps in time and only a poor moving image can be obtained.
The present invention is addressed to solve this problem.
Further, as the digital signal processing technique improves, many image sensing apparatuses adopting digital recording and reading technique in the recording and reproducing unit have been proposed. In these image sensing apparatuses, image signals are compressed and encoded as well as modulated to a format suitable for digital recording in a recording unit, then recorded in a data storage medium. Further, when reproducing image signals, read data is demodulated and decoded in a process in opposite to the recording process, then outputting reproduced image signals.
FIG. 11 is a block diagram illustrating a configuration of a conventional image sensing apparatus. In FIG. 11, reference numeral 501 denotes an image sensing unit whose focus, zoom ratio, and iris diaphragm, and so on, are controlled by an image sensing controller 503, and which generates known digital standard image data S1p, such as parallel data conforming with SMPTE (Society of Motion Picture and Television Engineers) 125M.
A block division unit 502 divides the digital image data S1p into blocks consisting of a plurality of pixels, further applies processes, such as shuffling and noise reduction, on the divided digital image data.
The image data S2p divided into a plurality of blocks by the block division unit 502 is provided to a motion detector (MD) 505.
The MD 505 generates information S3p on movement of image data on the basis of the input image data S2p and outputs it to a system controller 509. The MD 505 detects movement in an image by detecting differences between field image data of each image blocks.
Reference numerals 506 and 507 denote discrete cosine transform (DCT) units which compress information by using correlation between neighboring pixels of the image data. The first DCT unit 506 performs DCT on image data by an area, e.g., 8×8 pixel block, of a frame image.
Further, the second DCT unit 507 performs DCT on image data by an area, e.g., by 8×4 pixel block, of an odd line field image, and 8×4 pixel block of an even line field image.
The system controller 509 outputs a switching signal S4p in accordance with information on movement, thereby controls a switch 508 to switch between the first DCT unit 506 and the second DCT unit 507.
Here, in a case where no movement is detected in the block image data S2p, in other words, movement determination information S3p shows “not moving”, the switch 508 switches to the terminal 508a. In contrast, in a case where movement is detected in the block image data S2p, i.e., the movement determination information S3p shows “moving”, the switch 508 switches to the terminal 508b. Thus, DCT processes can be switched for a frame image and for a field image in accordance with the movement determination information S3p. 
In a case where there is a large movement in the block image data S2p, since correlation between fields is low, vertical components of the DCT coefficients reach high frequency range if an image is processed as a frame image, and encoding efficiency drops extremely.
Therefore, when the movement determination information S3p shows “moving”, it is controlled so that odd line field image data and even line field image data are separately applied with orthogonal transformation. As described above, by properly switching the DCT between the one for a frame image and the one a field image in accordance with a state, “moving” or “not moving” shown by the movement determination information S3p, effective encoding is performed.
Data S5p processed with the DCT by the first DCT unit 506 or the second DCT unit 507 is quantized by a quantization unit 510. Processes at each step of quantization are adjusted in accordance with the precision of the image data, and an image of low frequency is quantized closely, whereas an image of high frequency is quantized roughly.
This is because distinguishable ability of human eyes is keen for an image of low frequency, in contrast, it is dull for an image of high frequency. Therefore, by quantizing image data of low frequency range closely and image data of high frequency range roughly, distortion of an image caused by the quantization is concentrated on the high frequency components, thereby reducing deterioration of a visual image quality.
An encoding unit 511 scans the block data arranged in two dimension in zig-zag scanning from the low space frequency portion to the high space frequency portion to obtain linear data, encodes zero coefficients by run-length coding and non-zero coefficients by two dimensional Huffman coding into variable length codes, then outputs encoded data S6p. 
In the run-length coding, image data is applied with lossless compression in accordance with a zero-run count. In the Huffman coding, short codes are assigned to data whose occurrence probability is high, whereas long codes are assigned to data whose occurrence probability is low, thereby shortening the total code length.
A flag controller 512 is for generating a system information flag S7p used when writing information outputted from the system controller 509 to a recording medium.
In an image sensing apparatus adopting digital recording and reading technique, not only a moving image but also a still image can be recorded in high precision. Further, it is possible to record a still image of high precision (still image recording mode) while recording a moving image (moving image recording mode) depending upon an image sensing mode.
The conventional image sensing apparatus as described above determines movement based on correlation of block image data between fields, a proper DCT method performed by frame or by field is selected on the basis of the detected result.
However, upon sensing and recording a still image of high precision by using a progressive or non-interlace scanning type CCD, since information on the vertical resolution of the image contains higher frequency components than that of an image sensed by using an interlace scanning type CCD, it becomes very difficult to determine movement. For example, when a fine stripe pattern is in an image and differences between even line field data and odd line field data of the image are calculated to be used for detecting movement, since the differences between the field data would be large because of the high vertical resolution, there would be more chance for the image to be misjudged as a moving image. Accordingly, there are more cases in which a still image is misjudged as a moving image, thereby encoding efficiency drops.
The present invention is also addressed for solving the above problem.